I think we may want the option to connect to an existing Prometheus container so the control plane can be run in a container without needing docker in docker mounts

I think we may want the option to connect to an existing Prometheus container so the control plane can be run in a container without needing docker in docker mounts

That's fine, but the control plane needs to be able to write to the Prometheus config file wherever the Prometheus is, so currently having it in Docker only is the most feasible

https://prometheus.io/docs/guides/file-sd/

the Prometheus instance will listen for changes to the file and automatically update the scrape target list

scrape_configs:
- job_name: 'node'
  file_sd_configs:
  - files:
    - 'targets.json'

allows for targets.json to have

[
  {
    "labels": {
      "job": "node"
    },
    "targets": [
      "localhost:9100"
    ]
  }
]

this looks more like what we want https://github.com/prometheus/pushgateway

this looks more like what we want https://github.com/prometheus/pushgateway

I think we would have to modify snarkOS to make this work since they use a scrape exporter by default. The metrics exporter they use does support push gateways though

agent can scrape the node locally (with env + own node id context), then push it to a desired service

Closing as we've decided to not make the control plane spin up its own Prometheus instance and instead let agents optionally use push gateway to a Prometheus server specified by the control plane.